Chris Patrick writes that SLAC researchers have built a neural-network method for compressing large scientific datasets while preserving fine details that conventional compression erases. The approach uses wavelet analysis to separate data features by scale, then encodes each scale separately through a neural network, enabling 10- to 100-fold file size reductions and selective decompression of only the regions a researcher needs.
- Published in Nature Machine Intelligence (August 24, 2026)
- Motivated by upcoming LCLS upgrades that will generate nearly one terabyte of data per second
- Tested successfully on X-ray diffraction data, solar magnetic field measurements, and photographs
- Neural networks trained on Perlmutter at NERSC (Lawrence Berkeley National Lab)
- Co-developers include researchers from UC Davis and Carnegie Mellon University
- Published in Nature Machine Intelligence (August 24, 2026)
- Motivated by upcoming LCLS upgrades that will generate nearly one terabyte of data per second
- Tested successfully on X-ray diffraction data, solar magnetic field measurements, and photographs
- Neural networks trained on Perlmutter at NERSC (Lawrence Berkeley National Lab)
- Co-developers include researchers from UC Davis and Carnegie Mellon University
OpenZL is a new open source data compression framework that offers lossless compression for structured data, achieving performance comparable to specialized compressors by applying configurable transforms to reveal hidden order in the data.
